Methods and systems for controlling applications using user interface device with touch sensor

ABSTRACT

The functionality of a conventional mouse is extended to provide an extended number of simultaneously adjustable user interface parameters employing one or more user-removable modules. In an embodiment, a user interface for controlling an external device, such as a computer, includes a first user interface sensor configured with a housing. This first sensor generates a first plurality of signals responsive to movement of the housing relative to two orthogonal axes. A compartment is configured with the housing and is sized to receive the user-removable module. This user-removable module contains a second user interface sensor, which generates a second plurality of signals responsive to user manipulation. Output is provided responsive to signals generated by the first and second user interface sensors. In another embodiment, the housing of an extended functionality mouse itself serves as a module removable from a compartment provided in another physical device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/728,730, filed Apr. 25, 2022, which is a continuation of U.S. patentapplication Ser. No. 15/209,188, filed Jul. 13, 2016, now U.S. Pat. No.11,314,340, which is a continuation of U.S. patent application Ser. No.14/469,453, filed Aug. 26, 2014, now U.S. Pat. No. 9,417,716, which is acontinuation of U.S. Patent application Ser. No. 11/008,892, filed Dec.10, 2004, now U.S. Pat. No. 8,816,956, which is a continuation-in-partof U.S. patent application Ser. No. 10/806,694, filed Mar. 22, 2004, nowabandoned, which is a continuation of U.S. patent application Ser. No.10/779,368, filed Feb. 13, 2004, now U.S. Pat. No. 7,620,915, thedisclosures of which are incorporated herein in their entirety.

BACKGROUND

User interface devices for data entry and graphical user interfacepointing have been known for many years. The most common devices includethe computer mouse (usually attributed to English, Engelbart, and Berman“Display-Selection Techniques for Text Manipulation, IEEE Transactionson Human Factors in Electronics, pp. 5-15, vol. HFE-8, No. 1, March1967), the trackball, the touchpad in both finger-operated (for example,the various finger-operated devices produced by Symantec Corp., ofSpringfield, Oregon) and stylus-operated (for example, products usedwith desktop workstation computers—Wacom Technology Corp., of Vancouver,Washington) versions, and display-overlay touchscreens. Other historicaland exotic devices include various types of light pens and the DataGlove™ (produced by VPL Research, Inc., of Redwood City, California).

Most user interface devices for data entry and graphical user interfacepointing commonly used with computers or with equipment providingcomputer-like user interfaces have two wide-range parameter adjustmentcapabilities that are usually assigned to the task of positioning ascreen cursor within a two-dimensional display. In many cases, one, two,or three binary-valued “discrete-event” controls are provided, typicallyin the form of spring-loaded push-buttons.

More recently, computer mice have emerged that provide an additional“scroll” finger-wheel adjustment (for example, between two controlbuttons) to provide a third wide-range parameter adjustment capability(for example, various products developed by Logitech Inc., of Fremont,California). A mouse of this configuration is often referred to as a“Scroll Mouse” since this third wide-range parameter is typicallyassigned the task of positioning a vertical scroll bar in an activelyselected window. This additional finger-wheel adjustment may alsooperate as a spring-loaded push-button, thus providing an additionalbinary-valued “discrete-event” control. Typically this additionalbinary-valued “discrete-event” control is used to turn on and off anautomatic scrolling feature which controls the rate and direction ofautomatic scrolling according to vertical displacement of the displayedcursor.

SUMMARY

In an embodiment, the functionality of a conventional mouse is extendedto provide an extended number of simultaneously adjustable userinterface parameters employing one or more user-removable modules. In anembodiment, a user interface for controlling an external device, such asa computer, includes a first user interface sensor configured with ahousing. This first sensor generates a first plurality of signalsresponsive to movement of the housing relative to two orthogonal axes. Acompartment is configured with the housing and is sized to receive theuser-removable module. This user-removable module contains a second userinterface sensor, which generates a second plurality of signalsresponsive to user manipulation. Output is provided responsive tosignals generated by the first and second user interface sensors. Inanother embodiment, the housing of an extended functionality mouseitself serves as a module removable from a compartment provided inanother physical device.

Other embodiments of the disclosure include a freely-rotating trackballfor simultaneously detecting one, two, or three independent directionsof its non-rotational displacement, and as many as three independentdirections (roll, pitch, and yaw) of its rotation. In variousimplementations, non-rotational displacement of the trackball may bemeasured or interpreted as a widely-varying user interface parameter oras a discrete “click” event. Signal processing may be used to derivethree independent rotation components (roll, pitch, and yaw) from moreprimitive sensor measurements of the trackball. The disclosure providesfor trackball displacement and rotation to be sensed by a variety ofsensing techniques including optical, magnetic, electromagnetic,capacitive, resistive, acoustic, resonance, and polarization sensor. Thesystem may be used to provide an extended number of simultaneouslyinteractive user interface parameters, and may itself be incorporatedinto larger user interface structures, such as a mouse body.

In accordance with embodiments of the disclosure, a traditionalhand-movable computer mouse is configured with an additional userinterface sensor. For convenience, the term “user interface sensor” willbe used herein to collectively refer to devices such as trackballs,touchpads, mouse devices, scroll-wheels, joysticks, and other suchdevices.

In one aspect of the disclosure, the addition of a user interface sensorprovides alternative physical modalities for the same pair of adjustableparameters so that a user may switch between using the user interfacedevice as a traditional hand-movable computer mouse and using the userinterface device as a trackball or touchpad.

In another aspect of the disclosure, the addition of a user interfacesensor provides alternative resolution modalities for the same pair ofadjustable parameters so that a user may switch between using anembodiment as a traditional hand-movable computer mouse to obtain onelevel of parameter adjustment resolution, and using the embodiment as atrackball or touchpad, for example, to obtain a different level ofparameter adjustment resolution.

In another aspect of the disclosure, the addition of a user interfacesensor provides alternative types of warping modalities for the samepair of adjustable parameters so that a user may switch between using anembodiment as a traditional hand-movable computer mouse to obtain onetype of parameter adjustment (for example, linear) and using theembodiment as a trackball or touchpad, for example, to obtain adifferent type of parameter adjustment (for example, logarithmic,gamma-corrected, arccosine, exponential, etc.).

In another aspect of the disclosure, the addition of a user interfacesensor provides alternative offset modalities for the same pair ofadjustable parameters so that a user may switch between using anembodiment as a traditional hand-movable computer mouse to obtain onetype of centering of parameter adjustment and using the embodiment as atrackball or touchpad, for example, to obtain a different centering ofparameter adjustment.

In another aspect of the disclosure, the addition of a user interfacesensor may be used to provide additional parameters that may besimultaneously controlled.

In another aspect of the disclosure, the addition of a user interfacesensor may be used to provide additional parameters that are of adifferent isolated context from those assigned to a traditionalhand-movable computer mouse.

In a further more detailed aspect of the disclosure, the addition of atouchpad may be used to provide many additional parameters that are of adifferent context than those of a traditional hand-movable computermouse.

In a further more detailed aspect of the disclosure, the touchpad may bea null-contact touchpad adapted to measure at least one maximum spatialspan of contact in a given direction.

In a yet further detailed aspect of the disclosure, the null-contacttouchpad is adapted to measure at least one maximum spatial span ofcontact in a given direction at a specifiable angle.

In an additional further detailed aspect of the disclosure, thenull-contact touchpad is adapted to measure pressure applied to thenull-contact touchpad.

In a further more detailed aspect of the disclosure, the touchpad maycomprise a pressure sensor array touchpad adapted to measure, amongother things, one or more of the following: the rocking position of acontacting finger in a given direction; the rotational position of acontacting finger; the pressure of a contacting finger; and parametersrelating to a plurality of contacting fingers.

In another aspect of the disclosure the addition of a user interfacesensor may be realized via a replaceable module accepted by anadaptation of a traditional hand-movable computer mouse. In thisimplementation, a user may initially obtain an embodiment in oneconfiguration and field-modify it to another configuration.

In another aspect of the disclosure, a traditional hand-movable computermouse may be implemented as a removable module in a laptop computer orother affiliated equipment, and may include a wireless link with thelaptop computer or other affiliated equipment.

In yet a further aspect of the disclosure, a traditional hand-movablecomputer mouse is implemented as a removable module in a laptop computeror other affiliated equipment, and the mouse further comprises a userinterface sensor.

In another aspect of the disclosure, a traditional hand-movable computermouse additionally comprises a trackball or touchpad, for example. Inthis aspect, the mouse comprises a wireless link to an associatedcomputer or other affiliated equipment.

In another aspect of the disclosure, a visual display is provided.

In another aspect of the disclosure, auditory output is provided.

In another aspect of the disclosure, two or more individual userinterface sensors may be combined without incorporation of such sensorswith a traditional hand-movable computer mouse.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the presentinvention will become more apparent upon consideration of the followingdescription of preferred embodiments taken in conjunction with theaccompanying drawing figures, wherein:

FIGS. 1 a-1 i illustrate various implementations involving merging,selecting, multiplexing, and preprocessing distributed in various waysbetween the body of the user interface device and an associated piece ofequipment;

FIGS. 2 a-2 c depict an embodiment of the disclosure comprising atraditional mouse fitted with a trackball, illustrating three exemplarybutton configurations;

FIGS. 3 a-3 c depict an embodiment of the disclosure comprising atraditional mouse fitted with a touchpad, illustrating three exemplarybutton configurations;

FIGS. 4 a-4 d depict various degrees of freedom that may be measurablyassigned to a trackball for interactively controlling parameters in auser interface;

FIG. 4 e depicts a device with a trackball and sensors;

FIGS. 5 a-5 d depict various degrees of freedom that may be measurablyassigned to a touchpad for interactively controlling parameters in auser interface;

FIG. 6 depicts an illustrative implementation of the disclosure directedtowards the control of both a traditional text cursor and adual-scrollbar in a typesetting application;

FIG. 7 depicts an illustrative implementation of the disclosure directedtowards the active selection from a clip-art or symbol library andadjustment of positioning or other attributes of the active selection ina drawing or layout application;

FIG. 8 is a flowchart showing illustrative operations and overheadinvolved in selecting and adjusting a specific pair of parameters fromamong a larger group of adjustable parameters;

FIGS. 9 a-9 b illustrate how the illustrative operations and overheaddepicted in FIG. 8 introduce excessive overhead in situations where manyparameters with a larger group of adjustable parameters must be adjustedin pairs;

FIGS. 10 a-10 b illustrate one technique for adding an additionalscroll-wheel to a conventional scroll-wheel mouse;

FIGS. 11 a-11 b illustrate a simple example of open adjustments beingmade within various levels of hierarchy of graphical object groupings;

FIG. 12 illustrates aspects of the 3D orientation of an object in3-dimensional space, and in particular the three coordinates of positionand the three angles of rotation;

FIGS. 13 a-13 b illustrate one technique for using two cursors in a textcut-and-paste operation;

FIGS. 14 a-14 d illustrate illustrative embodiments of a mouse where thetraditional mouse buttons have been replaced by trackballs or touchpads;

FIG. 15 shows an illustrative implementation of a removable mouse modulefor use in conjunction with a laptop computer;

FIG. 16 a shows a stylized version of a removable mouse module;

FIG. 16 b shows the removable mouse module of FIG. 16 a inserted into alaptop computer;

FIG. 17 a shows another stylized version of a removable mouse module;

FIG. 17 b shows the removable mouse module of FIG. 17 a inserted into alaptop computer; and

FIGS. 18 a-18 d show various control arrangements which may beconfigured with a mouse.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawing figures which form a part hereof, and which show by way ofillustration specific embodiments of the disclosure. It is to beunderstood by those of ordinary skill in this technological field thatother embodiments may be utilized, and structural, electrical, as wellas procedural changes may be made without departing from the scope ofthe claimed subject matter.

By way of overview, a number of different applications that takeadvantage of the functionality of additional, wide-range adjustmentparameters will now be discussed. In one example, an additionalfinger-wheel adjustment device providing a third, wide-range parameteradjustment capability is typically assigned to vertical scroll barpositioning. In accordance with the disclosure, such a design may besupplemented with a fourth, wide-range parameter adjustment capabilityso that a horizontal scroll bar position control may be achieved. Withthe increasing popularity of the web (with many web pages wide enough torequire horizontal scrolling) and publisher layout tools (typicallyinvolving pages wide enough to require horizontal scrolling), as well asthe need for simultaneous interactive horizontal and vertical scrollingactions that do not disturb a screen cursor location when using “zoom”controls, a fourth wide-range parameter adjustment capability intraditional user interface devices for data entry and graphical userinterface pointing is quite valuable.

There are many other potential uses for additional wide-range adjustmentparameters in traditional user interface devices for data entry andgraphical user interface pointing. Some opportunities have wide-rangeapplicability, such as in providing interactive separate adjustment ofthe selections for “cut” or “copy” operations from the interactiveadjustment of insertion location or selection for a “paste” operation.Other opportunities are more specialized but still widely applicable,such as making an active selection from a clip-art or symbol library andadjusting the position or other attributes of said active selection in adrawing or layout application. Yet other opportunities may be veryspecialized, such as in 3D modeling, data visualization, advanced coloradjustment, lighting control, machine control, or audio and image signalprocessing.

There are many opportunities for adjusting the same two widely-varyingparameters in more than one way. For example, one user interfacemodality (such as normal mouse operation) may be used for normalparameter adjustment, while a second user interface modality may be usedfor adjustments involving a different resolution, warping (i.e.,logarithmic, gamma-corrected, arccosine, exponential, etc.), centeringoffset, etc. Another important case is where the same two widely-varyingparameters are controlled with the same resolution, warping, offset,etc., but in a different user interface modality (e.g., a trackball ortouchpad may have some advantages in certain situations over use of atraditional mouse). A more widely applicable example is that ofresponding to and preventing hand/wrist/arm fatigue and injury. Atraditional mouse fitted with an additional user interface sensor allowsa user to interchangeably enter information with either the mouse bodyor another user interface sensor, changing which user interface modalityis used (obtaining the same results with either) to relieve fatigue orpain, or prevent injury.

More specifically, the addition of a user interface sensor provides manyopportunities for alternative means of adjustment of a common pair ofadjustable parameters. The user may benefit from having both adjustmentmodalities available, changing modalities as needed or desired. Forexample:

-   -   A user may simply switch between using an embodiment as a        traditional hand-movable computer mouse and using the embodiment        as another kind of user interface sensor.        -   The user may benefit from having both modalities available            to avoid or in response to hand fatigue.        -   The user may also benefit from having both modalities            available due to the type of pointing or data entry            interaction needed—depending on the case, one type of            modality may perform better than another.    -   The trackball, touchpad, or other user interface sensor        apparatus may be used to provide alternative resolution        modalities so that a user may switch between using an embodiment        as a traditional hand-movable computer mouse to obtain one level        of parameter adjustment resolution and using the embodiment as a        user interface sensor to obtain a different level of parameter        adjustment resolution.    -   The trackball or touchpad may be used to provide alternative        warping modalities for the same pair of adjustable parameters so        that a user may switch between using an embodiment as a        traditional hand-movable computer mouse to obtain one type of        parameter adjustment (for example, linear) and using the        embodiment as another kind of user interface sensor to obtain a        different type of parameter adjustment resolution (for example,        logarithmic, gamma-corrected, arccosine, exponential, etc.).    -   The user interface sensor may be used to provide alternative        offset modalities for a common pair of adjustable parameters so        that a user may switch between using an embodiment as a        traditional hand-movable computer mouse to obtain one centering        of parameter adjustment and using the embodiment as another kind        of user interface sensor to obtain a different centering of        parameter adjustment. These modalities can provide one or more        “location bookmarks” for cursor location, each affiliated with a        sub-context within an interactive application.

Further, the addition of another user interface sensor provides manyopportunities for the simultaneous adjustment of additional parametersthat may or may not require simultaneous interactive control. Thetraditional computer mouse may be used to simultaneously adjust twoparameters while the additional user interface sensor may be configuredto allow the fingers to simultaneously adjust at least two additionalparameters. In some applications, these additional parameters may beclosely related to those assigned to the traditional computer mouse. Forexample, the traditional computer mouse may be used to simultaneouslyadjust the location within a window of a text, graphic, or other object,while the additional user interface sensor allows the fingers to be usedto adjust the type or attributes of the text, graphic, or other object.In other applications, these additional parameters may be of a differentisolated context from those assigned to the traditional computer mouse.For example, the traditional computer mouse may be used tosimultaneously adjust two parameters dealing with affairs within anactive application window, while the addition of another user interfacesensor allows the fingers to be used to adjust at least two additionalparameters dealing with broader window system affairs such as verticaland horizontal scrollbars, window selection, window resizing, etc., orintermediate-level affairs such as zoom control, help-window navigation,clip-art selection, etc. Another application would be to provideseparate adjustment of selections for “cut” or “copy” operations fromthe adjustment of insertion location or selection for a “paste”operation.

In instances of the disclosure involving the addition of a touchpad, thetouchpad may be configured and/or enhanced to allow the fingers toadjust three or more additional interactive measured parameters. Theseadditional interactive measured parameters may be assigned to controlmore sophisticated interactive affairs such as 3-dimensional spaceposition, 3-dimensional space orientation, color model navigation, imageor audio processing parameter settings, etc.

The additional interactive measured parameters (above the two typicallyassociated with traditional touchpads) may be provided in a number ofways. For example, the touchpad may be a relatively low-costnull-contact touchpad that has been adapted to measure at least onemaximum spatial span of contact in a given direction. The user may alsocontrol an additional parameter by varying the width between the spatialextremes of a single point of contact (i.e., how much finger flesh makescontact with the pad) or multiple points of contact (i.e., the spreadbetween two contacting fingers). As there are two geometricallyorthogonal sensing directions on a touchpad, this provides the user witha method for controlling four total parameters from a touchpad. Further,rotational transformations or other methodologies may be used to measurethe angle of rotation of an oblong contact profile. The measured anglemay be used as a fifth interactive parameter, and/or used to adapt themeasurement of maximum spatial span of contact in an arbitrary angle.The null-contact touchpad may be further adapted to measure pressureapplied to the null-contact touchpad via, for example, use of anattached pressure sensor. The pressure may be used as a sixthinteractive parameter, and may or may not have rapid pressure changesrecognized as ‘tap’ or ‘click’ events.

Another way to provide the additional interactive measured parameters(above the two typically associated with traditional touchpads) with atouchpad is to implement the touchpad with a pressure sensor array.Through use of operations effectively amounting to image processing, apressure sensor array touchpad can be adapted to measure the rockingposition of a contacting finger in two orthogonal directions, as well asthe rotational position and average pressure of a contacting finger.Thus a pressure sensor array touchpad can be adapted to provide up tosix widely variable interactive adjustable parameters from the contactof a single finger. A pressure sensor array touchpad can be furtheradapted to measure parameters relating to a plurality of contactingfingers.

All of these considerations and others demonstrate the potential valuein providing the addition of another user interface sensor to atraditional hand-movable computer mouse. In the descriptions to follow,various implementations and illustrative applications of illustrativeembodiments are considered and explained.

1. Illustrative Signal Flow and Processing

The disclosure provides for a wide range of signal flow and processingconfigurations and implementations. FIGS. 1 a-1 i illustrate variousillustrative implementations involving merging, selecting, multiplexing,and preprocessing distributed in various ways between the body of theuser interface device and an associated piece of equipment. FIGS. 1 a-1d concern the aggregated pair of user interface sensors in isolation,while FIGS. 1 e-1 i address arrangements where some functions areperformed in the associated external equipment. It is noted that thedisclosure further provides for any of these illustrativefunctionalities, as well as other functionalities, to be combined ormade selectable. In any of the illustrative implementations disclosedherein, power may be supplied to these implementations by the associatedexternal equipment or by other devices such as batteries, storagecapacitors, photoelectric devices, and the like.

FIG. 1 a shows an implementation 100 a featuring two user interfacesensors 101, 102, each of which may be a particular type of userinterface sensor (which again may be a trackball, touchpad, mouse, orother user interface device) that can be collocated within the commonphysical enclosure 100 (demarcated by the dotted-line boundary). Firstuser interface sensor 101 produces signal 108 and second user interfacesensor 102 produces another signal 109, which are directed to a merge orselect function 103. The merge or select function produces outgoingsignal 110 which is provided to associated external equipment. Heresignals 108, 109 (from first and second user interface sensors 101, 102)would typically lose their individual identities within the outgoingsignal 110 and as such may be used or processed interchangeably (withoutindividual attribution to either the first or second user interfacesensor) by the associated external equipment.

Merge or select function 103 may take several forms in variousimplementations. For example, in one embodiment it may simply be fixedto only perform a merge operation. In another embodiment it may onlyprovide a selection function; here the selection function may becontrolled by the user using a switch or some sort of action, or theselection may be remotely controlled by external equipment. As analternative, merge or select function 103 may instead provide a useradjustable “merge” or “select” function.

FIG. 1 b shows an implementation that is similar to that of FIG. 1 a .The primary difference is that the FIG. 1 b implementation 100 breplaces merge or select function 103 of FIG. 1 a with multiplexfunction 104 to produce outgoing signal 110. Here signals 108, 109 (fromfirst and second user interface sensors 101, 102) retain theirindividual identities within outgoing signal 110 and as such may be usedor processed separately by the associated external equipment.

FIG. 1 c shows implementation 100 c which is similar in many respects tothat of FIG. 1 a . However, the FIG. 1 c embodiment utilizespreprocessor 105 applied to signal 109 to produce processed signal 109a. The processed signal 109 a, along with signal 108, is directed tomerge or select function 103, resulting in outgoing signal 110.Preprocessor 105 may thus introduce a pre-processing step (such asresolution modification, warping modification, offset modification,etc.) on signal 109 to produce a signal of distinguished value from thatof signal 108.

FIG. 1 d illustrates another illustrative implementation 100 d which issimilar to that of FIG. 1 c but with an additional preprocessor 106applied to signal 108. Preprocessor 106 produces processed signal 108 a,which along with signal 109 a, is directed to merge or select function103 to produce outgoing signal 110. Preprocessor 106 may thereforeintroduce a pre-processing step (such as resolution modification,warping modification, offset modification, etc.) on signal 108 toproduce a signal of either equivalent or distinguished value from thatof signal 109 a.

FIG. 1 e shows implementation 100 b of FIG. 1 b (which features two userinterface sensors 101, 102 and a multiplex function 104 producing anoutgoing signal 110) used in conjunction with subsequent functionsprovided by associated external equipment 150 a. These subsequentfunctions are shown within the functional boundary 150 of the associatedexternal equipment 150 a.

Outgoing signal 110 from the common physical enclosure 100, is presentedto demultiplexer 117 within external equipment 150 a. Demultiplexer 117produces signal 118 corresponding to or associated with pre-multiplexedsignal 108, and an additional signal 119 corresponding to or associatedwith the pre-multiplexed signal 109. Here, signals 118, 119 arepresented to merge or select function 113 producing merged or selectedsignal 120. This implementation is functionally similar or equivalent tothat of FIG. 1 a except that the various types of merge or selectionfunctions 103 are provided within the associated external equipment 150a (for example in software, perhaps within an application where it iscustomized for the needs of that application) rather than being providedwithin physical unit 100 b.

FIG. 1 e shows implementation 100 b of FIG. 1 b used in conjunction withsubsequent functions provided by associated external equipment 150 b,and similar to that of FIG. 1 e , except signal 119 produced bydemultiplexer 117 is directed to preprocessor 115 to produce processedsignal 119 a before being sent to merge or selection function 113. Thisimplementation is thus functionally similar or equivalent to that ofFIG. 1 c except that the various types of merge or selection 103 andpreprocessor 105 functions in FIG. 1 c are provided within theassociated external equipment 150 b (for example in software, perhapswithin an application where it is customized for the needs of thatapplication) rather than being provided within the physical unit 100 b.

FIG. 1 g shows implementation 100 b of FIG. 1 b used in conjunction withsubsequent functions provided by associated external equipment 150 c.This arrangement expands on that shown in FIG. 1 f in that signal 118produced by demultiplexer 117 is directed to preprocessor 116 to produceprocessed signal 118 a. The processed signal is then sent to merge orselection function 113. This implementation is thus functionally similaror equivalent to that of FIG. 1 d except that the various types of mergeor selection 103 and preprocessor 105, 106 functions in FIG. 1 d areprovided within the associated external equipment 150 c (for example insoftware, perhaps within an application where it is customized for theneeds of that application) rather than being provided within thephysical unit 100 b.

FIG. 1 h shows implementation 100 b of FIG. 1 b used in conjunction withsubsequent functions provided by associated external equipment 150 d.Here again, as in the arrangement of FIG. 1 f , signal 119 produced bydemultiplexer 117 is directed to preprocessor 115 to produce processedsignal 119 a. In contrast to other embodiments, processed signal 119 ais not directed to merge or selection 113 function and retains itsidentity for use by a different destination from that of signal 118within associated external equipment 150 d.

As a final illustrative example in this series, FIG. 1 i showsimplementation 100 b of FIG. 1 b used in conjunction with subsequentfunctions provided by associated external equipment 150 e. Here, as inthe arrangement of FIG. 1 g , signals 118, 119 produced by demultiplexer117 are directed to preprocessors 115, 116 to produce processed signals118 a, 119 a. In contrast to other embodiments, processed signals 118 a,119 a are not directed to merge or selection 113 function and thusretain their identity for use by differing destinations withinassociated external equipment 150 e.

Having presented various illustrative signal flows and processingrealizations, attention is now directed to illustrative implementationsutilizing specific types of user interface sensors. It is to beunderstood that the various sensors, techniques and methods disclosedherein may be implemented using computer software, hardware, andfirmware, and combinations thereof.

2. Implementations Utilizing Specific Types of Additional User InterfaceSensors

In this section, a number of illustrative implementations are set forthutilizing various types of additional user interface sensors added to anoriginal user interface sensor or device. The first three sectionsaddress cases where the original user interface sensor is a movablemouse and the additional user interface sensor is a trackball, touchpad,or other illustrative technology, including additional scroll-wheels.Then illustrative adaptations of trackballs and touchpads, eachtraditionally used to provide simultaneous adjustment of two interactivewidely-varying parameters, are extended to provide simultaneousadjustment of as many as six interactive widely-varying parameters andother forms of control. This section continues by presentingillustrative implementations where the original user interface sensor isnot a mouse, where there are a plurality of additional user interfacesensors, where there is a visual display or auditory output, where theadditional user interface sensor is a removable module, and where theimplementation itself is a removable module.

2.1 Trackball Implementations of Additional User Interface Sensors

FIGS. 2 a-2 c illustrate a number of implementations where a trackballcontroller is used as an additional user interface sensor apparatusincorporated into a traditional hand-movable computer mouse. In each ofthese implementations it is understood that the trackball may be freelyoperated without disturbing previous or currently-varying parameteradjustments made by the mouse.

In one implementation, a trackball controller is added to the topsurface of a conventional computer mouse as depicted in FIG. 2 a . Theconventional mouse buttons may be located in various places in view ofthe presence of the trackball and in synergy with it. In theconfiguration depicted in FIG. 2 a , buttons 201 and 202 are located onthe surface of mouse 200; button 201 being on the left of the trackballand button 202 being on the right of the trackball.

In a second configuration depicted in FIG. 2 b , buttons 231 and 232 arenow separated and located on the sides of mouse 230 as is the case withmany trackball interfaces; button 231 being on the left side of mouse230 and button 232 being on the right side of the mouse.

In a third possible configuration depicted in FIG. 2 c , elongatedbuttons 261 and 262 are shown located on the surface of the mouse 260;button 261 wraps around the left of the trackball and button 262 wrapsaround the right of the trackball. The elongated buttons 261 and 262 maybe positioned so that a user can readily and rapidly move fingers fromtrackball 265 to buttons 261 and 262, or even operate one of thesebuttons with one finger while another finger contacts trackball 265.

It is noted that unlike the touchpad described below, the trackball hasan effectively unconfined range of contiguous data entry.

2.2 Touchpad Implementations of Additional User Interface Sensor

FIGS. 3 a-3 c illustrate a number of illustrative implementations wherea touchpad controller is used as an additional user interface sensorincorporated into a traditional hand-movable computer mouse. In each ofthese implementations it is understood that the touchpad may be freelyoperated without disturbing previous or currently-varying parameteradjustments made by the mouse.

In one implementation, trackball 205 in FIG. 2 a can be replaced withtouchpad 305 as shown in FIG. 3 a . Additionally, this touchpadimplementation can also support the alternative button configurations ofFIGS. 2 b and 2 c . By way of illustration, FIG. 3 b shows buttons 331,332 positioned on either side of the mouse body 330, while FIG. 3 cshows elongated buttons 361, 362 on the surface of the mouse wrappingaround either side of touchpad 365.

It is noted that unlike the trackball, the touchpad typically has aconfined maximum range of data entry by contiguous operation of afinger, stylus, etc.

2.3 Other Implementations of Additional User Interface Sensors

In accordance with other embodiments, the disclosure provides for stillother types of additional user interface sensors. In each of theseembodiments it is understood that any of these user interface sensorsmay be freely operated without disturbing previous or currently varyingparameter adjustments made by the mouse or other associated device.

As a first example, an X-Y joystick may be used in place of thetrackball or touchpad described above. The joystick may have aspring-return or may retain the last position it was placed. Similar tothe touchpad and unlike the trackball, the X-Y joystick typically has aconfined maximum range of travel.

As another example, two or more scrolling finger wheels may be used inplace of the trackball or touchpad described above. The scrolling fingerwheels may be implemented with an unconfined maximum range of travelsimilar to the trackball, or with a confined range of travel like thetouchpad and X-Y joystick. In this embodiment, it may be advantageous tohave one or more finger scroll-wheels mounted with its adjustmentdirection perpendicular to that of another finger scroll-wheel so thateach wheel may be appropriately associated with vertical and horizontalscroll bars of a window, or other useful orthogonally-based userinterface metaphor. For example, looking ahead to FIGS. 10 a and 10 b ,embodiments 1000 are depicted comprising the usual components of ascroll-wheel mouse (mouse body 1001, buttons 1011 and 1012, and theusual scroll wheel 1021) complemented with an additional scroll wheel1022 with adjustment direction perpendicular to that of fingerscroll-wheel 1021.

As another example, two or more rotating control knobs may be used inplace of the trackball or touchpad described above. Like the scrollingfinger wheels, the rotating control knobs may be implemented with anunconfined maximum range of travel like the trackball or with a confinedrange of travel like the touchpad and X-Y joystick.

The disclosure also provides for more exotic types of user interfacesensor technologies—for example, proximity detectors of various types(RF, infrared, etc.), video cameras (using any of the techniquesprovided in U.S. Pat. No. 6,570,078), and other emerging and unforeseentechnologies—to be used as the additional user interface sensoraggregated in the same physical enclosure as the first user interfacesensor.

2.4 Larger Numbers of Interactively Widely-Varying Adjustable Parametersfrom the Additional User Interface Sensor

In accordance with embodiments of the disclosure, additional userinterfaces may be used to capture larger numbers (i.e., more than two)of widely-varying adjustable parameters from the additional userinterface sensor. Some examples are provided here, but many others arepossible as may be readily understood by one skilled in the art.

FIGS. 4 a-4 d address the case of the trackball. FIG. 4 a illustrates afreely-rotating trackball 205 and the two principle orthogonaladjustment directions 401, 402 that are responsively resolved andmeasured in traditional trackball user interface devices. However, atleast two other physical degrees of freedom may be readily exploited,and at least six total parameters can be interactively adjusted andmeasured. FIG. 4 b shows the application of downward pressure 403 ontrackball 205. Such pressure 403 may be applied without disturbingcurrent values established in orthogonal adjustment directions 401, 402.Further, the trackball may be implemented so that downward pressure 403may be applied while simultaneously adjusting trackball 205 inorthogonal adjustment directions 401, 402, particularly if the signalproduced by the measurement of downward pressure 403 incorporates amodest “grace” zone of non-responsiveness for light pressure values.Downward pressure impulses may alternatively be sensed and treated asdiscrete event “taps,” as commonly used in contemporary touchpadinterfaces found in laptop computers, for example.

FIG. 4 c shows the application of “yaw” rotation 404 (i.e., rotationaround the vertical axis) of trackball 205. This yaw rotation 404 may beapplied without disturbing current values established in orthogonaladjustment directions 401, 402 and can be readily measured and adjustedas a widely-varying parameter. Further, by grasping trackball 205 (orother operational methods), the yaw rotation 404 and traditionalorthogonal adjustment directions 401, 402 may be independently andsimultaneously adjusted. It is also noted that in principle up to sixwidely-varying physical degrees of freedom can be simultaneouslymeasured from a properly configured trackball 205 by placing thetrackball 205 in a cradle that senses not only downward pressure 403 ordisplacement but also lateral pressure or displacement. As shown in FIG.4 d , both forward-backward, non-rotational force 405 and left-right,non-rotational force 406 may be applied to the trackball in a mannerthat the values of force or displacement in each of these directions405, 406 can be independently measured. Thus, by grasping trackball 205(or other operational methods), three rotational directions oforientation 401, 402, 404 and three non-rotational directions of forceor displacement 403, 405, 406 may be independently and simultaneouslyadjusted by a user and measured as six independent interactivelyadjustable user interface parameters. These correspond, effectively, tomeasurable adaptations of the six degrees of freedom of an orientableobject in 3-dimensional space as found in classical mechanics andaeronautics—that is:

-   -   “roll” rotation (adapted to 401)    -   “pitch” rotation (adapted to 402)    -   “up-down: displacements (adapted to 403)    -   “yaw” rotation (adapted to 404)    -   “forward-backward” displacements (adapted to 405)    -   “left-right” displacements (adapted to 406).

Most trackball sensing technologies use optically based techniques forsensing the two traditional components of rotation (“roll” and “pitch”)of the trackball. Trackball 205 itself may be configured with an opticalpattern on it with spatially varying reflectivity for a range of thelight spectrum. The pattern may be such that it can spatially vary lightreflectively in these two traditional components of trackball rotation.Alternatively, two spatially varying reflectivity patterns, each activeat different ranges of the light spectrum or light polarization, may besuperimposed or integrated with the trackball.

A number of approaches may be used to obtain measurements for all threedirections of rotation. In one completely optical approach, a second orthird spatially varying reflectivity pattern active at, respectively, asecond or third portion of the light spectrum (or light polarization ifavailable) may be superimposed or integrated with the patterns employedfor traditional “roll” and “pitch” rotation sensing, and an additionaloptical source and sensor 420 is used to obtain measurement of the addedvarying reflectivity pattern, as shown in FIG. 4 e . Depending on thepattern(s) used, sensor signals 421 may be directly usable or mayrequire processing of the three primitive signals measured by thesensors to obtain a clean decomposition of the, measurement signals intoindependent “roll,” “pitch,” and “yaw” signals independently responsiveto the “roll,” “pitch,” and “yaw” components of trackball rotation.

As another alternative, trackball 205 may include internally, or on itssurface, or both, materials with spatially varying patterns of magneticproperties, capacitive properties, electromagnetic properties,ultrasonic acoustic properties, resonance phenomena of any of theseproperties, polarization phenomena of any of these properties, etc.,individually or in combination, each of which may be active at specificranges of polarization, frequencies, etc. These may be used togetherwith or in place of optical measurement approaches. Again, depending onthe pattern(s) used, sensor signals may be directly usable or mayrequire processing of the three primitive signals measured by thesensors to obtain a clean decomposition of the measurement signals intoindependent “roll,” “pitch,” and “yaw” signals independently responsiveto the “roll,” “pitch,” and “yaw” components of trackball rotation as isclear to one skilled in the art. It is also noted that the thirdcomponent of rotation of the freely-rotating trackball may beinterpreted or even measured as a discrete “click” event.

Similarly, a number of approaches may be used to obtain measurements forone, two, or three directions of non-rotational trackball displacement.For example, trackball 205 may be secured in saddle 415, typicallyattached in some manner to housing 410 (for example, mouse 200, 230,260), allowing free rotation of trackball 205 but causing anydisplacement actions on the trackball to invoke displacements of saddle415. The saddle displacement may be measured with displacement sensor430 generating displacement signals 431. Displacement sensor 430 maycomprise one or more pressure, optical, resistive, capacitive, magnetic,electromagnetic, continuous-range sensors, switches, etc. It is alsonoted that one or more components of displacement of the freely-rotatingtrackball may be interpreted or even measured as a discrete “click”event.

FIGS. 5 a-5 d turn now to the case of the touchpad. FIG. 5 a illustratestouchpad 305 and the two principle orthogonal data entry directions 501,502 that are responsively resolved and measured in traditional touchpaduser interface devices. The touchpad shown in FIGS. 5 a-5 d , whichprovides at least four other physical degrees of freedom, may beimplemented using the techniques presented in U.S. Pat. No. 6,570,078,for example.

FIG. 5 b illustrates the use of downward pressure 503 in the context ofa touchpad. In contemporary touchpad interfaces, such as those found inlaptop computers for example, such downward pressure 503 is sensed andutilized as discrete event “taps.” However, downward pressure 503 mayalso be measured and adjusted as an independent and simultaneouslyinteractive widely-varying parameter. Further, as illustrated in FIG. 5c , the rotational angle 504 of a finger contacting a touchpad withrough-elliptical contact boundary can also be measured as awidely-varying parameter. In FIG. 5 d , both forward-backward 505 andleft-right 506 components of the tilt of a contacting finger canadditionally be measured as independent and simultaneously interactivewidely-varying parameters.

The sensing of multiple fingers, the application of contact syntaxes andgrammars, and other user interface control expansions of an adequatelyconfigured touchpad may also be achieved using, for example, thetechniques presented in U.S. Pat. No. 6,570,078.

The disclosure also provides for larger numbers (i.e., more than two) ofwidely-varying adjustable parameters from other types of user interfacesensor technologies. In the case of an X-Y joystick, the joystick may beconfigured to rotate on its axis, pulled in and out, fitted with a knobor trackball, etc., in a measurable fashion to provide additional andsimultaneous interactively adjustable parameters. In the cases of fingerscroll-wheels and rotational knobs, three or more of these devices maybe provided. When implementing video cameras, known techniques for theextraction of additional parameters may be used. Examples of the varioustypes of video extraction techniques that may be used are presented inU.S. Pat. No. 6,570,078.

2.5 Non-Mouse User Interface Sensors

In one of its most abstract forms, the disclosure provides for theincorporation of two conventional user interface sensors (such as amouse, trackball, touchpad, joystick, etc.) into an embodiment where theuser may freely use both of the user interface sensors individually orsimultaneously with the same hand. As such, the disclosure provides forimplementations that do not include a mouse as one of the user interfacesensors. For example, two or more individual user interface sensors canbe combined without need of a traditional hand-movable computer mouse.Such an implementation may be accomplished by implementing one of thepossible user interface sensors in place of the traditional hand-movablecomputer mouse where taught in other descriptions of the disclosure.This configuration may be useful when built into a laptop computer,control console, musical instrument, and test instrument, among others.

In one illustrative implementation of a non-mouse embodiment, atrackball and touchpad may be arranged in the same physical enclosure sothat the front, middle, or back of the palm may freely operate aconventional trackball while one or more selected extended or archingfingers may simultaneously or alternatively operate a touchpad. In thisexample, the touchpad may be a conventional touchpad providing twowidely-varying simultaneously interactive parameters from a singlefinger contact, or the touchpad may be a more enhanced version providingas many as six widely-varying and simultaneously interactive parametersfrom a single finger contact. The touchpad may also be configured toaccept multiple points of contact, recognize gestures, support syntaxand grammatical constructions using, for example, the teachings providedby U.S. Pat. No. 6,570,078.

In another non-mouse implementation, two trackballs may be arranged inthe same physical enclosure. In one possible arrangement, the twotrackballs may be positioned so that they lie parallel to the length ofthe hand, enabling the front, middle, or back of the palm to freelyoperate a first trackball while one or more extended or arching fingersmay simultaneously or alternatively operate the second trackball.

In another arrangement, the two trackballs may be positioned so thatthey lie parallel to the width of the hand, so that the fingers and/orthumb on the left side of the hand may operate a leftmost trackballwhile the remaining fingers and/or thumb on the right side of the handmay individually or simultaneously operate a rightmost trackball. Ineach of these arrangements, either or both of the trackballs may be aconventional trackball providing two widely-varying and simultaneouslyinteractive parameters, or it may be a more enhanced trackball providingas many as six widely-varying and simultaneously interactive parametersas described earlier.

In addition to the just-described embodiments, alternative arrangements,such as the combination of a palm-operated trackball and a recessedjoystick, and others, are also provided for by the disclosure.

2.6 Use of More than One Additional User Interface Sensor

Typically the arrangements of two non-mouse user interface sensorsdescribed above in Section 2.5 can also be applied to embodiments of thedisclosure where a mouse user interface sensor is used. In suchembodiments, a mouse user interface sensor is supplemented with at leasttwo additional user interface sensors ergonomically arranged so that thetwo additional user interface sensors may be simultaneously oralternatively operated by the same hand. If these embodiments arefurther configured so the mouse body is readily moved with adequateprecision via the back of the operating hand, then all three userinterface sensors may be simultaneously or alternatively operated by thesame hand in an ergonomically advantageous manner.

FIGS. 14 a-14 d illustrate some illustrative embodiments of thejust-described features. FIG. 14 a illustrates a mouse where traditionalmouse buttons have been replaced by trackballs 1405 a, 1405 b. Thesetrackballs 1405 a, 1405 b may accept a downward-pressure impulse and assuch act as the traditional mouse buttons, However, trackballs 1405 a,1405 b are also adjustable and each readily provides two or moreadditional widely-variable and simultaneously adjustable parameters inaddition to the two parameters adjusted by moving the mouse body 1400.

FIG. 14 b shows a similar arrangement where traditional mouse buttonshave been replaced by touchpads 1435 a, 1435 b. If desired, thesetouchpads may accept a downward-pressure impulse and as such act astraditional mouse buttons, but are also adjustable as touchpads and assuch each readily provides two or more additional widely-variable andsimultaneously adjustable parameters. As described in Section 2.5, asingle hand may be positioned to comfortably operate simultaneously oralternatively both trackballs or both touchpads. If these embodimentsare further configured so the mouse body is readily moved with adequateprecision via the back of the operating hand, then either of theseembodiments readily provides six to twelve widely-variable andsimultaneously adjustable parameters.

Other configurations are of course possible. For example, theconfigurations of FIGS. 14 a and 14 b may be blended as depicted in FIG.14 c , or in its minor image. As another example, FIG. 14 d illustratesa more extreme realization comprising a left-fingers/thumb trackball1465 a, a right-fingers/thumb trackball 1465 b, a palm trackball 1465 c,and a traditional clickable scroll-wheel 1468. Yet another alternativeis to replace one or more of the trackballs of the FIG. 14 d embodimentwith a touchpad user interface sensor.

2.7 Incorporation of Visual Display and Auditory Output

If desired, any of the mouse and non-mouse embodiments may furtherinclude a visual display. The visual display may provide details ofadjustable parameter values, operation modalities, etc. The visualdisplay may be physically associated with a physical enclosure (such asthat of a traditional computer mouse), or may be displayed on thecomputer screen or display of other associated equipment.

Alternatively or additionally, any of the mouse and non-mouseembodiments may further provide auditory output. The auditory output mayprovide details of adjustable parameter values, operation modalities,error conditions in usage of an embodiment, a condition relating toelapsed time or other metric of consistent use of a single usagemodality, etc. The auditory output associated with an embodiment may bephysically associated with a physical enclosure (such as that of atraditional computer mouse), or may be produced by speakers or headsetsaffiliated with the computer or other associated equipment.

2.8 Provisions for Field Installation or Replacement of Additional UserInterface Sensor

The disclosure also provides for the user interface sensor to beimplemented using a replaceable module accepted by an adaptation of atraditional computer mouse. In this implementation a user may initiallyobtain an embodiment in one configuration and field-modify it to anotherconfiguration.

2.9 Implementation as a Module Removable from Affiliated Equipment

The disclosure also provides for a traditional computer mouse to beimplemented as a removable module in a laptop computer or otheraffiliated equipment, and may include a wireless link with such devices.In particular, this removable module may further include one or moreuser interface sensors, with these sensors operable as a traditionaltrackball or touchpad when stowed in the laptop computer or otheraffiliated equipment in such a way that the traditional hand-movablecomputer mouse modality is unmovable and hence unusable.

FIG. 15 shows an illustrative implementation of a removable mouse modulefor use in conjunction with a laptop computer. The illustrative standardformat laptop computer 1500 shown in the figure comprises a keyboardarea 1501, a non-keyboard area 1502, as well as a screen area 1503comprising an LCD, OLED, plasma, or other type of visual display. Morespecific to this illustrative implementation, the non-keyboard area 1502comprises a cavity 1505 in the area which most contemporary laptopcomputers use an embedded touchpad. This cavity is such that it accepts,retains and locks into place the removable module 1550 which is shownhere comprising a touchpad 1552 as well as buttons 1553, 1554. Otherconfigurations, for example additional buttons, fewer buttons, atrackball or other user interface sensor in place of the touchpad 1552,different layouts of buttons and touchpads, different styling, etc. maybe used as would be clear to one skilled in the art.

In the illustrated embodiment, the removable module 1550 operates atleast as a traditional mouse when removed from the cavity 1505 andplaced in a movable fashion on a sufficiently level surface. In thisexample the buttons may be usable in the same way regardless whether theremovable module 1550 is secured within the cavity 1505 or removed fromthe cavity as depicted in FIG. 15 . Similarly, the touchpad 1552 wouldbe operable in the usual fashion when the removable module 1550 issecured within in the cavity 1505 and may be operable, disabled, oruser-selectably disabled in types of various implementations whenremoved from the cavity 1505 and used as a traditional mouse, forexample.

The removable module 1550 may transfer and/or exchange data with thelaptop computer 1500 by either a wireless link or an electrical cable.If an electrical cable is used, the removable module may receiveelectrical power through this cable. If a wireless link is used, theremovable module may include a rechargeable power source such as abattery, high-capacitance capacitor, etc. in which case recharging maybe performed automatically when the removable module is reinserted intothe cavity 1505. Alternatively, the removable module 1550 may internallyinclude replaceable batteries. The removable module 1550 mayadditionally or alternatively include one or more photoelectric cells toprovide power as is relatively commonplace in many contemporarycalculators.

The wireless link between the removable module 1550 and the laptopcomputer 1500 may be optical, radio, etc. The “mouse” user interfacesensor may be a rolling ball but may advantageously be implemented usingoptical mouse technology to facilitate reduced thickness of the body1551 of the removable module 1550 in association with the required depthof the cavity 1505 and thickness of the closed laptop computer. It thisexample it is to be understood that in some applications the touchpadmay not be needed and the removable module 1500 may only function as atraditional mouse when removed and placed on a surface. It this exampleit is also to be understood that the laptop computer 1500 canalternatively be any other type of equipment (test equipment, fieldinstrumentation, control panels, service panels, etc.) benefiting fromthis removable module implementation.

FIG. 16 a shows an illustrative stylized version 1650 of a removablemodule aspect of the disclosure as described above. This versionfeatures a more rounded body 1651, and in keeping with the removablemodule 1550 of FIG. 15 , again is illustrated comprising a touchpad 1652and buttons 1653, 1654. However, other configurations, for example withadditional buttons, fewer buttons, a trackball or other user interfacesensor in place of the touchpad 1652, different layouts of buttons andtouchpad, different styling, etc. may be used as would be clear to oneskilled in the art.

FIG. 16 b shows an example of how the stylized removable module 1650could be secured within an associated cavity 1605 (akin to the cavity1505 depicted in FIG. 15 ) in the larger body of a laptop computer,control panel, etc. 1600. The laptop computer, control panel, etc. 1600is shown here with a keyboard/control area 1601 and non-keyboard/controlarea 1602 in keeping with the laptop computer depiction of FIG. 15 , buta wide range of other configurations are possible. The stylizedremovable module 1650 could be released for removal, or evenappropriately ejected, from the cavity by operating a locking/releasetap, button, lever, or switch 1610.

In FIG. 16 b this locking/release tap, button, lever, or switch 1610 isshown at one edge of the body of a laptop computer, control panel, etc.1600 but could be located elsewhere and take on other forms and sizes(rounded button, rotating lock, segmented numerical combination lock, apair of buttons that must be operated simultaneously, etc.).Alternatively, the removable module 1650 could be released for removal,or even appropriately ejected, from the cavity under software control inresponse to keys, commands, combinations, passwords, etc. provided bykeys, controls, etc. within the larger body of a laptop computer,control panel, etc., a remote control, keycard reader, etc. Under thesecircumstances the stylized removable module 1650 may be latched andreleased by electromagnetic, piezo, or other electro-mechanical means.

FIG. 17 a illustrates another illustrative stylization 1750 of theremovable module implementation aspect of the disclosure that is inkeeping with the increasingly standard rounded-corner rectangular format1751 for touchpads in contemporary Asian-manufactured laptop computers.Again, a touchpad 1752 and buttons 1753, 1754 are shown but otherconfigurations, for example with additional buttons, fewer buttons, atrackball or other user interface sensor in place of the touchpad 1752,different layouts of buttons and touchpad, different styling, etc. maybe used as would be clear to one skilled in the art.

FIG. 17 b shows an example of how the stylized removable module 1750could be secured within an associated cavity (akin to the cavity 1505depicted in FIG. 15 ) in the larger body of a laptop computer 1700. Thelaptop computer 1700 is shown having keyboard area 1701 and non-keyboardarea 1702, although other configurations are possible. This stylizedremovable module 1750 could be released for removal, or evenappropriately ejected, from the cavity 1705 by operating a exemplarytwo-element locking/release tap, button, lever, or switch 1710 a, 1710b. Again, this locking/release tap, button, lever, or switch arrangement1710 a, 1710 b could be located elsewhere and take on other forms andsizes (rounded button, rotating lock, segmented numerical combinationlock, a pair of buttons that must be operated simultaneously, etc.), oralternatively, the removable module 1750 could be released for removal,or even appropriately ejected, from the cavity under software control inresponse to keys, commands, combinations, passwords, etc. provided bykeys, controls, etc. within larger body of a laptop computer, controlpanel, etc., a remote control, keycard reader, etc. Under thesecircumstances the stylized removable module 1750 may be latched andreleased by electromagnetic, piezo, or other electro-mechanical means.It is noted that the latching mechanism, electrical configuration, andother aspects (such as the wireless link) may implemented in such afashion that a composite module comprising the removable module 1750,the locking/release tap, button, lever, or switch arrangement 1710 a,1710 b, and interface connections to the rest of the laptop computer1700 may be configured so that said composite module exactly matches thesaid increasingly standard rounded-corner rectangular format fortouchpads in contemporary Asian-manufactured laptop computers. With theresulting interchangeable electrical and mechanical characteristics,both new and existing laptop computers with touchpads of thiscontemporary Asian-manufactured style may be optionally fitted orreadily upgraded to provide the removable module 1750 in place of thefixed touchpad.

3. Illustrative Applications

Departing now from the range of extreme realization and embodiments ofthe disclosure, attention is directed towards particular applications. Anumber of examples of various embodiments of the disclosure in a widerange of applications will now be presented. Many of these applicationsare viable with only the simplest physical embodiments (for example,those suggested by FIGS. 2 a-2 c and 3 a-3 c ). In the discussion thatfollows, particular note is directed towards the discussion in Section3.3 involving FIGS. 8 and 9 a-9 b. Although the discussion is motivatedby a graphical layout application, the principles of the discussion inSection 3.3 involving FIGS. 8 and 9 a-9 b are very general, and thediscussion illustrates the power of the disclosed subject matter forvarious applications in almost directly quantifiable terms.

3.1 Wrist/Hand/Arm-Fatigue Relief and Prevention Application

The danger and damage stemming from extensive continuous or mis-posturedmouse usage to wrist, hand, and arms are sadly misfortunate andincreasingly well recognized. As the present disclosure provides aplurality of different user interface sensors, it is well suited for usein responding to and preventing wrist/hand/arm fatigue due to overuse.In one illustrative implementation, user interface parameters can beinterchangeably adjusted with either the movement of the mouse body orthe use of an integrated trackball or touchpad with identical taskresults. Thus a user with a tiring hand can change at will the userinterface sensor employed according to how the hand feels or the natureof a specific task. In addition, to prevent fatigue or injury, the usercan also switch back and forth between moving the mouse body and usingthe trackball/touchpad either by free choice or by following auditory orvisual prompting from a time or usage monitor.

3.2 Double-Scrollbar Application

Contemporary mice often feature a small rotating wheel between thebuttons for use in operating the vertical scroll bar of a window withoutchanging the position of the mouse. In one particular application, theleft-right sensing capability of the trackball or touchpad may be usedto add a similar capability for horizontal scroll bars of a window.

In a trackball implementation, a user can move the vertical bar 611 ofFIG. 6 up by rotating trackball 205 away from him/herself, or one canmove the vertical bar 621 down by rotating trackball 205 towardshim/herself. By rotating trackball 205 to the left, the user can movethe horizontal bar 621 left. Similarly, the user can move the horizontalbar 621 right by rotating the trackball 205 to the right.

In a touchpad implementation, a user can move scroll bar 611 up bysliding the finger away from her/himself or move scroll bar 611 down bysliding the contacting finger towards her/himself; similarly, the usercan move the scroll bar 621 left by sliding the finger to the left ormove the scroll bar 621 right by sliding the contacting finger to theright. In another implementation, the vertical and horizontal scrollbars may be adjusted with a conventional scroll-wheel mouse that hasbeen fitted with an additional scroll-wheel. FIGS. 10 a and 10 b depictillustrative embodiments 1000 of such an arrangement which comprise theusual components of a scroll-wheel mouse including mouse body 1001,buttons 1011 and 1012, and traditional scroll wheel 1021, with thesecomponents further complemented by an additional scroll wheel 1022 withadjustment direction perpendicular to that finger scroll-wheel 1021.FIG. 10 a illustrates an arrangement where the additional scroll-wheelis located closer to the user while FIG. 10 b illustrates an arrangementwhere the additional scroll-wheel is located farther away from the user.In each of these arrangements, the two scroll-wheels are shownco-centered with respect to the mouse body; for simultaneous adjustmentit may be advantageous to locate the additional scroll-wheel 1022 to oneside or the other of the centered positions shown in FIGS. 10 a and 10 b. One approach useful for supporting both left-handed and right-handedusers, which may provide additional utility, would be to provide twooff-centered additional scroll-wheels, one on either side of the centerline of the mouse body 1001 and conventional scroll-wheel 1021.

3.3 Traditional 2D Layout, CAD, and Graphics Applications

In most contemporary 2-dimensional layout and graphics applications,such as those commonly used for viewgraphs, page layout, electronic CAD,etc., numerous mouse operations are necessary for each of the many typesof object attribute modification, etc. Typically, these mouse operationsare required because the mouse only allows for the interactiveadjustment of two widely-varying parameters at a time, and the user mustchange context several times as the parameters adjusted by the mouse arechosen, adjusted, and then replaced with another pair of parameters. Thepresent disclosure is useful in many of these circumstances because itallows for more than two parameters to be adjusted at the same time.

FIG. 7 shows an example of a session involving the authoring of aviewgraph. The viewgraph authoring task showcased in this exampleincludes the creation of a flowchart diagram (here depicting a businessworkflow process) and as such also illustrates related needs andattributes of a 2D CAD program involving layout of a diagram (such as acircuit, algorithm, etc.) or physical object (such as a PC board,control panel, semiconductor photolithography mask, etc.). In that thisexample further involves drawing, the example also illustrates therelated needs and attributes of a paint-box or electronic draftingapplication.

In this broadly representative application, application window 700 isshown comprising menu area 700 a and drawing area 700 b. Within thedrawing area, viewgraph title 701 and portion 702 of the flowchart to bedrawn, comprising thus far a sequence of flowgraph objects connected byarrows, have already been entered and rendered. New flowgraph objectsmay be introduced in standard fashion by selecting the type of newobject desired from a palette, initially putting an instance of theselected object type in a convenient place in the drawing area,adjusting the size, orientation, color, and/or other attributes, andputting into final position. Often a number of the last few steps areinteractively cycled through multiple times before the newly introducedobject is adequately drawn and the user directs their attention to thenext task. In this example, the palette of available objects of aspecific high-level task is shown as an overlapping stack of threesub-class palettes 703 a, 703 b, 703 c, each providing a selection ofavailable objects within that sub-class. Here, for example, sub-classpallet 703 a has been selected (as indicated by the heavy line) fromother available objects 705, within the sub-class pallet 703 a. Uponselection, an initial highly adjustable rendering of a specific instance714 of the selected object 704 appears in a convenient location, whichmay be selected by the user.

The specific instance 714, rendered in this highly adjustable initialstate, is typically surrounded by graphical handles 716 which facilitatesizing, positioning within the drawing area, and often at leastrotational orientation (for example, using the mouse with the ALT keysimultaneously held down to interactively adjust the angle of rotationof the object should that be needed). Traditionally, the cursorcontrolled by the mouse 717 can be moved within object 714 to relocateit within drawing area 700 b or can, as shown in FIG. 7 , be positionedatop one of the graphical handles 716 to permit the mouse to adjust thehorizontal and vertical scale of object 714, i.e., adjust its size andaspect-ratio. In some application packages, the latter adjustment ispermitted to collapse the object through to ‘zero’ thickness in one ofthe adjustment dimensions and continue through to re-render the objectin mirror image, thus additionally providing a form of vertical andhorizontal flip by using the size and aspect-ratio resizing.

As familiar and widely accepted as these sorts of operations are, thereis considerable overhead involved in such sequences of repeatedselecting and adjusting (and in some cases additional deselecting) pairsof parameters from a larger collection of parameters. To see severalaspects of the power of the present disclosure, these operations are nowexamined in more detail in generalized form.

FIG. 8 is a flowchart showing tasks involved in selecting and adjustingone of a plurality of available pairs of adjustable parameters by usinga user interface device permitting the adjustment of only one pair ofparameters at a time. In FIG. 8 , the task goal is simply to adjust apair of selected parameters 801 from a larger group of adjustableparameters. However, since a larger number of parameters are availablefor adjustment than are available at one time with the user interfacesensor, the specific pair of parameters must first be selected. In mostknown graphical user interface systems and methods, this typicallyinvolves first using the user interface device to control the movementof a cursor to a selection area of the graphical interface in a firstoverhead step 811 and then selecting the adjustment context (parameterpair) in a second overhead step 821.

In some situations the selected pair of parameters may be immediatelyadjusted in goal operation 801, but typically the cursor must then atleast be moved, in a third overhead operation 812, to a location outsideof the selection area affiliated with operations 811 and 821 to anotherlocation (such as a drawing or typing area) affiliated with the context(parameter pair) that has just been selected for adjustment. In somesituations the selected pair of parameters may be immediately adjustedin goal operation 801, but typically the context must be activated (forexample, by clicking in an open portion of a drawing area or selectingan existing object) in a fourth overhead operation 822.

After the selected pair of parameters are adjusted (for example, bysizing a rectangle, etc.) the cycle may then immediately be repeated insome variant form for another pair of parameters, but typically theparameters must be deselected (for example, by another click to set thefinal value) in a fifth overhead operation 823 before the cursor may bemoved to the selection area in another instance of operation 811. Insummary, in order to adjust one pair of parameters from a larger groupof parameters, as many as five overhead operations (as many as twocursor movements 811, 812 and as many as three select/deselect clicks821, 822, 823) are commonly required.

FIGS. 9 a-9 b show broader implications of the overhead called out inFIG. 8 . FIG. 9 a depicts the sequential adjustment of pairs ofparameters chosen from a larger group of pairs of parameters in ascenario suggestive of no interactive iteration. One pair of parametersis adjusted with up to five overhead operations in action 901, then asecond pair of parameters is adjusted with up to five overheadoperations in action 902, then a third pair of parameters is adjustedwith up to five overhead operations in action 903, and so on. Here theoverhead slows things down but may not be a significant encumbrance tothe broader goal of actions 901, 902, 903, etc.

In contrast, FIG. 9 b depicts an interactive adjustment of pairs ofparameters from a larger group of parameters in a scenario suggestive ofone where interactive iteration is required, as the setting of one pairof parameters is difficult to complete without setting other parameters.Here the overhead is likely a significant encumbrance to the higher goalinvolving the pair-wise adjustment actions 901, 902, 903, etc. Forexample, consider the interactive adjustment of six parameters, one pairat a time, through pair-wise adjustment actions 901, 902, 903: not onlyare up to five operations of overhead involved for each of the pair-wiseadjustment actions 901, 902, 903, but a considerable extra number ofpasses must be made through these pair-wise adjustment actions 901, 902,903 due to the fact that the adjustment of some parameters depends on orinteracts heavily with the values of other parameters. The situationgets even more cumbersome should additional pair-wise adjustmentoperations be required. FIG. 9 b further shows the potential for one ormore additional adjustment actions 950 which in principle may beiterated as well as and combined with pair-wise adjustment actions 901,902, 903 (as suggested by fully-connected iteration paths 921, 922,923). In contrast to such sequences or iterative graphs of pair-wiseadjustment actions, the present disclosure readily offers, for example,four, six, eight or even higher numbers of simultaneously adjustableparameters controllable by the same hand which, when selected in acontext, eliminate the many overhead operations depicted in FIGS. 8 and9 a-9 b, and the many additional iteration steps depicted in FIG. 9 b.

Returning now to the generalized graphical layout situation describedearlier and depicted in FIG. 7 , the following operations are routinelyperformed in 2D graphics, layout, and CAD applications:

-   -   A. Selection of palette containing object;    -   B. Selection of object from palette;    -   C. Selection of “layer” object is to be assigned to (common in        CAD, but typically not used in standard draw and paint        packages);    -   D. Adjustment of object placement in drawing;    -   E. Adjustment of object sizing;    -   F. Adjustment of object rotation;    -   G. Adjustment of object line thickness(es);    -   H. Adjustment of object line color(s);    -   I. Adjustment of object fill color(s); and    -   J. Adjustment of object fill pattern(s);

Of these, operations B, D, and E are almost always utilized, operationsA, G, and I are frequently utilized, and operations C, F, H, and J arerarely utilized.

Thus, in one illustrative application, it may be advantageous to groupspecific collections of operations that are commonly used together (thismay be application-specific) so that the benefits of having four or morewidely-adjustable interactive parameters simultaneously available can beapplied to speed the execution of basic common operations. For example:

-   -   Employing a four-parameter version:        -   Operation 1: Mouse for operation        -   Operation 2: Mouse for operation D, trackball or touchpad            for operation E    -   Employing a six-parameter version comprising a 4-parameter        touchpad:        -   Operation 1: Mouse for operation B, touchpad finger-location            for operation D, touchpad finger-tilt for operation E.    -   Employing a six-parameter version comprising a mouse and two        trackballs or touchpads:        -   Operation 1: Mouse for operation B, first trackball/touchpad            for operation D, second trackball/touchpad for operation E.

Other operations can be later applied in groupings and operationsappropriate for the application.

As a possible alternative to the preceding example, it may beadvantageous to assign a principal one of the user interface sensors tothe sequential adjustment of each of such universal (or otherwiseprincipal) operations and reserve the additional user interface sensorsfor rapid “in-context” interactive access to less frequently usedoperations. For example:

-   -   Employing a four-parameter version:        -   Operation 1: Mouse for operation B, trackball or touchpad            for operation A and/or operation C;        -   Operation 2: Mouse for operation D, trackball or touchpad            for operation E and/or operation F;        -   Operation 3: Mouse for operation G, trackball or touchpad            for operation H; and        -   Operation 4: Mouse for operation I, trackball or touchpad            for operation J.    -   Employing a six-parameter version comprising a 4-parameter        touchpad:        -   Operation 1: Mouse for operation B, touchpad finger-location            for operation A and touchpad finger-tilt for operation C;        -   Operation 2: Mouse for operation D, touchpad finger-location            for operation E and touchpad finger-tilt for operation F;        -   Operation 3: Mouse for operation G, touchpad finger-location            (and touchpad finger-tilt as useful) for operation H; and        -   Operation 4: Mouse for operation I, touchpad finger-location            (and touchpad finger-tilt as useful) for operation J.    -   Employing a six-parameter version comprising a mouse and two        trackballs or touchpads:        -   Operation 1: Mouse for operation B, first trackball/touchpad            for operation A and second trackball/touchpad for operation            C;        -   Operation 2: Mouse for operation D, first trackball/touchpad            for operation E and second trackball/touchpad for operation            F;        -   Operation 3: Mouse for operation G, first trackball/touchpad            (and second trackball/touchpad as useful) for operation H;            and        -   Operation 4: Mouse for operation I, first trackball/touchpad            (and second trackball/touchpad as useful) for operation J.

As another alternative example, the user may freely assign userinterface sensor parameters to operations A through J (and others as maybe useful) for each of a number of steps as may match the task or tasksat hand. These assignments may be stored for later retrieval and use,and may be named by the user. The stored assignments may be saved alongwith specific files, specific applications, or as a general template theuser may apply to a number of applications. It is noted that suchvariable assignments may be particularly useful to users as their handsfatigue, to prevent fatigue or injury, or as an adjustment for atemporary or permanent disability.

3.4 Multi-Resolution Mouse Application

In another illustrative family of applications, one user interfacesensor (for example, the mouse body) is used for course adjustment orfine adjustment of user interface parameters, while the additional userinterface sensor (for example, a trackball or touchpad) is used for theremaining level of parameter adjustment resolution.

In most user-interface applications it is advantageous to have multiplescales of graphical user interface pointing and data entry. Many windowsystems provide an ‘acceleration’ setting which changes the pointing anddata entry values on a more significant scale used for user interfacechanges made less frequently. Many applications further internallyadjust the resolution as the corresponding visual display is “zoomed” inand out.

In many user interface applications, additional levels of resolutionselection may be useful. For example, in pointing usage in text work,multiple resolutions would be advantageous in amending fine print or inmaking isolated changes in thumbnail overviews of 40% actual size orless. Similarly, in graphics work, fine resolution may be especiallyuseful in making fine adjustments to figures. In the fine adjustment offigures, it may be further advantageous to employ each of the separateuser interface sensors in conjunction with corresponding snap-grids ofdiffering grid spacing, particularly if one of the grid spacings is asub-multiple of the other. A potentially useful extension of this wouldbe to impose locally-applicable grid spacing on individual graphic orother objects, particularly objects which have been resized and hencefor which the standard snap-grid spacing is no longer useful.

In a further application, the user interface may be directed towardsnon-positional adjustments, such as the adjustment of a rotation angleor of the color of a graphic object; here multiple resolutions may bevaluable to make careful adjustments and coarse adjustments as needed.Similarly, scroll bars for long documents may also benefit from rapidaccess to multiple resolution scales, for example one user interfacesensor may be used to navigate within a page (using a fine-grainednavigation scale) while a second user interface sensor may be used tonavigate across pages (using a coarser-grained navigation scale).

3.5 Provision of Both Absolute and Relative Positioning

As discussed earlier, some types of user interface sensors, such as thetouchpad and X-Y joystick for example, naturally have a limited maximumrange of operation while others such as a mouse, trackball, andscroll-wheel have an essentially unlimited maximum range of operation.Although most user interface sensors are interpreted in relative terms(that is, the stimulus from the sensor is interpreted as a command tomove a cursor, scroll bar, etc., incrementally in some directionrelative to a current position), stimulus signals from any of thesetypes of user interface signals may be interpreted in either a relativeor absolute manner with varying degrees of naturalness or problematicqualities.

The present disclosure provides for one user interface sensor to be usedfor absolute positioning of a cursor, scroll bar, etc., or other meansof parameter adjustment while another user interface sensor is used fortraditional relative adjustment of such parameters. For example, ascroll bar may be adjusted in the usual fashion by a mouse body ortrackball and in an absolute manner by a touchpad wherein the extremevalues of the adjusted parameter correspond to the extreme positions atthe edges of the touchpad. In one embodiment or application settingthese two user interface sensors may control the same parameters—here itis often the result that the two sensors adjust the same parameters withdifferent resolutions. Further, in this situation it is fairly likelythat at least one of the resolution scale factors will be adjustedautomatically. For example, in a document editor, as the number of pagesof the document varies, the resolution of the absolute positioningsensor will correspondingly vary (so that the extremities in range of,for example, a touchpad correspond to the top of the first page and endof the last page) while the relative positioning sensor may retain thesame incrementing/decrementing vertical scrolling resolution scaleregardless of the number of pages.

3.6 Color-Selection Application

In color adjustment, three parameters are involved in the fullinteractive span of any complete color space (RGB, HSB, YUV, etc.). Byadding additional parameters to the overall user interface, all threeparameters can be adjusted simultaneously rather than simply two at atime. As the present disclosure provides at least four simultaneousinteractively adjustable parameters overall, it is thus potentiallyuseful for fully interactive color adjustment within a complete colorspace model. Further, should the additional user interface sensor besuch that it alone provides three simultaneously interactivelyadjustable parameters, the first user interface sensor (for example, themouse body) may be used as a pointer to select objects and theadditional user interface sensor may be used to adjust attributes of theselected object such as its color, border color, etc.

3.7 Multi-level Graphic Object Grouping and Editing Application

In many drawing applications, lower-level graphical or other objects(such as lines, basic shapes, and text areas) may be grouped to form anaggregated object. This aggregated or “grouped” object (collectivelyreferred to herein as an “aggregated object”) can then be moved,rotated, flipped, resized, etc. as if it were a lower-level graphic orother object. Grouping can also typically be done hierarchically and inmixed hierarchies, i.e., a plurality of lower-level graphical or otherobjects may first be grouped, and the resulting aggregated object maythen itself be grouped with other aggregated objects and/or lower-levelgraphical or other objects.

Often one or more of the lower-level graphical or other objectscomprising the aggregated object may need modification. In the case oftext, most applications permit modifications to be made to individualtext objects within an aggregated object. However, for any isolatedadjustment to any other lower-level graphical or other object theaggregated object must be first disaggregated or “ungrouped” tocompletely free the involved lower-level graphical or other object fromany grouping it had been involved in. After the modification, thegrouping must be reconstructed. Often this becomes a cumbersomesituation, particularly where the adjustments within the group arethemselves an interactive response to other adjustments made within adrawing.

The additional number of widely-adjustable simultaneously interactiveparameters made possible by the disclosure may be advantageously appliedto this problem. For example, one user interface sensor may be used tonavigate the levels of grouping and another user interface sensor may beused to perform operations on objects (lower level or “grouped”) withinthat level of grouping of the overall aggregated object.

As an illustrative example, FIG. 11 a shows a portion 1100 of a largerdrawing, the portion 1100 featuring box 1101, two arrowed lines 1111,1112, and grouped object 1102 (here itself comprising two trianglesconnected by a line). In this example it is given that grouped object1102 is itself grouped with box 1101 to form a second grouped object,and this second grouped object is itself grouped with the two arrowedlines 1111, 1112 to form a third grouped object. The user's task is tomodify FIG. 11 a so that it becomes FIG. 11 b . To do this, effectivelythe user must, in some order of operation:

-   -   Reposition grouped object 1102 from its original position in        FIG. 11 a to a new position in FIG. 11 b;    -   Copy or otherwise reproduce grouped object 1102 to create an        accompanying grouped object 1102 a, and position it within box        1101;    -   Introduce a vertically distributed ellipsis 1103 and position it        within box 1101—typically, a vertically distributed ellipsis        1103 is either rotated text or itself a fourth grouped object        created from three aligned text elements; and    -   Ensure elements 1102, 1102 a, and 1103 are in the end grouped        with box 1101 to form the second grouped object, and this second        grouped object is itself grouped with the two arrowed lines        1111, 1112 to form a third grouped object.

According to the disclosure, one user interface sensor is used to selectthe second group level, and the second user interface sensor is used toperform insert, copy, paste, and position operations within this levelof grouping without any form or type of ungrouping operation. If thevertically distributed ellipsis 1103 itself is realized as a fourthgrouped object created from three aligned text elements, when it ispasted into the drawing via this modality its 1103 grouping issubordinated appropriately (i.e., structured as a peer to groupedobjects 1102, 1102 a) within the second grouping level.

Although readily implemented using the novel user interface sensorsdescribed herein that make it particularly easy to simultaneously adjusta plurality of pairs of parameters, the aspects of the disclosureillustrated here can also be implemented with a conventional userinterface sensor such as a traditional mouse, trackball, or touchpad. Inthis case, the conventional user interface sensor such as a traditionalmouse, trackball, or touchpad must first be used to select the level ofgrouping and then be used to make the desired modifications within thatlevel of grouping; to make modifications at a different level ofgrouping, the new level of grouping must be selected in a separateoperation, thus adding overhead, as depicted in FIGS. 8 and 9 a-9 b.Although this novel ability to move and modify arbitrary graphic orother objects within groupings may be implemented in this way, having anadditional number of widely-adjustable simultaneously interactiveparameters—made possible by the main themes of the present disclosure—isclearly more efficient, as many or all of the overhead operationsdepicted in FIGS. 8 and 9 a-9 b can be eliminated via usage of theadditional widely-adjustable simultaneously interactive parameters.

3.8 3D Graphic Object Placement and Orientation Application

CAD and drawing packages that enable 3D object placement and orientationwithin a 3D space typically extend the capabilities of traditional 2Dlayout, CAD, and graphics applications as described in Section 3.3 toserve additional geometric needs. As illustrated in FIG. 12 , theplacement and orientation of 3D object 1200 within a 3D space 1250(oriented with respect to a reference point 1251) requires that oneadditional position dimension and two additional orientation angles bespecified to complete the full collection of three position dimensions1201, 1202, 1203 and three orientation angles 1211, 1212, 1213.

To interactively adjust these parameters pairwise (or individually witha knob box as has been done historically in some systems) involvescomplex repetitive passes among high-overhead operations as depicted inFIGS. 9 a-9 b , for example among steps 901, 902, 903. The necessity ofmaking many high-overhead operations, for example moving among steps901, 902, 903, can be functionally disruptive as well as slow andinefficient. The ability to interactively freely adjust the fullcollection of three position dimensions 1201, 1202, 1203 and threeorientation angles 1211, 1212, 1213 is thus of extremely high value.

The disclosure provides for a wide range of mappings between the sixposition and orientation parameters 1201, 1202, 1203, 1211, 1212, 1213involved in the placement and orientation of 3D object 1200 within a 3Dspace, and the large numbers of widely-adjustable and simultaneouslyinteractive parameters facilitated by various realizations of thedisclosure. As one example, a mouse fitted with two trackballs as inFIG. 14 a may be used to specify these six parameters in various ways.One technique is to use the position of mouse body 1400 to control twoof the position coordinates (for example 1202, 1203), one of thetrackballs (for example 1405 a) to control the orientation angles (1212,1213) corresponding to these two axes, and the remaining trackball (1405b) to control the remaining axis (1201) and its correspondingorientation angle (1211). In this example, trackballs 1405 a, 1405 b areconfigured or used in 2-parameter modalities.

In another example, the mouse of FIG. 14 a is fitted with twotrackballs, the trackballs may be configured in 3-parameter modalitieswith one of the trackballs used for controlling three positiondimensions 1201, 1202, 1203 and the second trackball configured tocorrespondingly control the three orientation angles 1211, 1212, 1213.Here the position of mouse body 1400 may be used to control otheraspects of drawing operations.

In another implementation, a touchpad configured for 4-parameteroperation involving two parameters of finger position and two parametersof finger tilt may be combined with a trackball configured for2-parameter operation. In this example, finger position may be used tocontrol two of the position coordinates (for example 1202, 1203), fingertilt may be used to control the orientation angles (1212, 1213)corresponding to these two axes, and the trackball to control theremaining axis (1201) and its corresponding orientation angle (1211). Ifthe configuration includes a mouse body, its position may be used tocontrol other aspects of drawing operations.

In another example, a configuration like that of FIG. 14 d may useleft-fingers/thumb trackball 1465 a to control a first positioncoordinate 1201 and its corresponding orientation angle 1211, theright-fingers/thumb trackball 1465 b to control a second positioncoordinate 1202 and its corresponding orientation angle 1212, and palmtrackball 1465 c to control third position coordinate 1203 and itscorresponding orientation angle 1213.

The disclosure further provides for a wide range of additional mappingsand geometric metaphors between the user interface sensor geometry andthe three position dimensions 1201, 1202, 1203 and three orientationangles 1211, 1212, 1213 of a 3D object.

3.9 Multiple Cursors and Cut and Paste Application

The disclosure additionally provides for a plurality of pairs of userinterface sensor parameters to be used to control the respectivepositions of a corresponding plurality of individual cursors,selections, and/or insertion points. Multiple cursors and associatedoperations of multiple selection and insertion points can have manyapplications. Below, a few of these possibilities that would be apparentto one skilled in the art are showcased in a cut-and-paste editingexample.

Cut, copy, and paste operations using traditional user interface devicesusually involve multiple operations to switch between contextsintroducing considerable overhead as depicted in FIGS. 9 a-9 b and 8.For instance, FIG. 13 a illustrates a text editing example with textdisplay window 1300 involving the selection of a clause 1320(highlighted in this example) with the intention of relocating it to anew position 1351. In such an operation with a traditional 2-parametermouse/trackball/touchpad user interface, the cursor is first used toselect clause 1320 and then used to select the insertion position 1351.

When writing or editing it is often the case that material needs to befetched from elsewhere and put in the spot where one is currentlywriting. Here, the cursor is initially in the spot where the insertionis to occur and the user must then lose the cursor position currentlyset in this spot to go searching and then to select and cut or copy thematerial to be pasted; following this the user must then search again,perhaps taking considerable time, for said initial spot and re-establishthe cursor location there. Equally often there are other situationswhere material must be split up and distributed in a number of far-flungplaces. Here, the cursor is initially in the spot where the material tobe sequentially divided and relocated is originally aggregated; the usermust repeatedly select the portion of the remaining aggregated materialto be relocated and then lose that cursor position to go searching forthe new destination insertion spot, perform the insertion, and thensearch again, perhaps taking considerable time, for the initial spot andre-establish the cursor location there. In both of these cases it wouldbe advantageous if the user could “bookmark” an initial cursor location,search and perform the desired fetch or relocation operations, andreadily return without search to the “bookmarked” location. Althoughthis novel and advantageously valuable capability could be realized witha conventional mouse through context redirection operations involvingthe steps depicted in FIGS. 8 and 9 a-9 b, the present disclosureprovides for a wide range of readily realized and easy-to-useimplementations.

An embodiment may be used in a minimal configuration capable ofinteractively specifying at least two pairs of widely adjustableinteractive parameters. Returning to the specific example associatedwith FIG. 13 a , one pair of parameters is used to set the location offirst cursor 1301 which is used in a selection operation, while thesecond pair of parameters is used to set the location of second cursor1351 which is to be used to independently set an insertion point. Theuser may then perform the cut and paste operation with a single mouseclick, resulting in the outcome depicted in FIG. 13 b . The relocatedtext clause 1320 has been transferred to a position determined by theinsertion cursor 1351 (here shown to the left of the cursor 1351; itcould just as easily be to the right of it), and cursor 1301 used tomake the selection remains in position. Either cursor 1301 or 1351 maynow be moved and/or used for other cut, copy, paste, or (via thekeyboard) new text insertion operations.

Although in this example the two cursor locations were close enough tobe displayed in the same window 1300, the value of this application issignificantly increased should the two positions be separated by manypages, many tens of pages, or even many hundreds of pages of text. Suchsituations may be handled by any number of approaches as is clear to oneskilled in the art. In one approach involving a single display window,the area comprising the cursor whose corresponding user interface sensorwas last manipulated is displayed in the single display window. Inanother approach involving a single display window, a click event orother user interface stimulus may be used to toggle among the areascomprising the various cursor locations. In yet another approach, atleast two windows may be rendered, with one of the cursors displayed andoperable within one window and a second cursor displayed and operable ina second window.

The disclosure also provides for these general principles to be appliedto other types of objects and applications, such as spreadsheet cells(involving data, formula objects, and cell formats), graphical objects,electronic CAD diagrams (where objects may be connected with formulas,dynamic models, etc.), and others as will be apparent to one skilled inthe art.

3.10 Simulation, Processing, and Analysis Applications

Simulation, processing, and analysis applications typically involve alarge number of parameters which are adjusted to model, affect orinvestigate the resulting behaviors, end results, and/or implications.Conventional 2-parameter user interface devices such as amouse/trackball/touchpad require the user to adjust these parameterspairwise (or individually with a knob box as has been done historicallyin some systems) involving complex repetitive passes among high-overheadoperations as depicted in FIGS. 9 a-9 b , for example among steps 901,902, 903. As in the case of 3D object positioning and orientation, thedivision among high-overhead operations, for example moving among steps901, 902, 903, can be functionally disruptive as well as slow andinefficient. The ability to interactively and freely adjust largercollections of parameters simultaneously is thus also of extremely highvalue.

3.11 Live Signal Processing and Lighting Applications

In artistic performance, composition, and recording applications,control of large numbers of parameters requiring simultaneousinteractive adjustment is common. Conventional recording, mixing, video,and light control consoles typically have large numbers of controls withcarefully designed spatial layouts to facilitate the rapid and preciseadjustment of multiple parameters via knobs, sliders, pushbuttons,toggle switches, etc. The introduction of computer GUIs has addedconsiderable value and new capabilities, including “soft” reconfigurableconsoles and functional assignments, but in the bargain typicallyencumber users—accustomed to rapid and precise operation of multipleparameters—with a 2-parameter mouse/trackball/touchpad having theoverhead of iterative context-switching operations depicted in FIGS. 8and 9 a-9 b. As in the case of 3D object positioning and orientation,the division among high-overhead operations, for example moving amongsteps 901, 902, 903, can be functionally disruptive as well as slow andinefficient. The ability to interactively freely adjust largercollections of parameters simultaneously is thus also of extremely highvalue in artistic performance, composition, recording mixing, video, andlight control applications. This is so much of an issue that newgenerations of generalized hardware “control surfaces” providingassignable sliders, switches, buttons, etc. have begun to appear (seefor example “High-End Control Surfaces” by Rob Shrock, ElectronicMusician, February 2002, pp. 72-80). In that various implementations ofthe present disclosure provide larger collections of simultaneouslyadjustable parameters, the disclosure provides a bridge between thetraditional mouse and the more complicated and expensive “controlsurface” hardware technologies. The present disclosure may be furtherspecialized to include special layouts of additional user interfacesensors—for example, as shown in FIGS. 18 a-18 b the top surface of amouse 1800 a, 1800 b may be outfitted with a number of finger operatedsliders 1811, narrowly columnated touchpads 1812, or other types ofcontrols pertaining to traditional contexts and layouts of controlconsoles in addition or as an alternative to the usual mouse buttons1801, 1802.

It is noted that, as shown in FIG. 18 c , narrowly columnated touchpadsfor use with individual fingers, functionally equivalent to that of FIG.18 b , can be realized with a single touchpad 1805 (as with FIGS. 3 a-3c ) that would be useful in more general settings; here it may beadvantageous to add an overlay bezel 1820 with finger slots 1830 andgraduation markings 1840 as shown in FIG. 18 d . Such a touchpad wouldoffer valuable utility if it separately sensed and resolved multiplepoints of contact and produced control data that could be subsequentprocessed to produce isolated control signals uniquely responsive toeach finger slot area of the touchpad.

3.12 Real-Time Machine Control and Plant Operations

Similarly, real-time machine control and plant (manufacturing, chemical,energy, etc.) operations also traditionally involve controlling asignificant number of parameters requiring simultaneous interactiveadjustment. Conventional real-time machine control and plant operationconsoles typically have large numbers of controls with carefullydesigned spatial layouts to facilitate the rapid and precise adjustmentof multiple parameters via knobs, sliders, pushbuttons, toggle switches,etc. The introduction of computer GUIs can add considerable value andnew capabilities, including “soft” reconfigurable consoles andfunctional assignments, but in the bargain typically significantlyencumber users—accustomed to rapid and precise operation of multipleparameters—with a 2-parameter mouse/trackball/touchpad having theoverhead of iterative context-switching operations depicted in FIGS. 8and 9 a-9 b. As in the case of 3D object positioning/orientation andartistic applications described earlier, the division amonghigh-overhead operations, for example moving among steps 901, 902, 903,can be functionally disruptive as well as slow and inefficient. Theability to interactively freely adjust larger collections of parameterssimultaneously is thus also of extremely high value.

A very few examples of this category of application where the disclosedsubject matter may be useful include many forms of robotics control,computer-control manufacturing tools, industrial optical and electronmicroscopy, camera control (pan, tilt, zoom, focus, and/or iris), plantprocess elements (heaters, pumps, values, stirrers, aerators, actuators,activators, etc.), and a wide range of other related and divergentpossibilities apparent to those skilled in the art.

3.13 Readily Available Zoom Control

In complex drawings, layouts, and other visual-interface applications itis often necessary to zoom in and out many times to adjust details atvarious levels of scale. In many user interfaces and applications thisinvolves extensive context switching overhead of the type depicted inFIGS. 8, 9 a, and 9 b. The additional interactively control parametersprovided for by the disclosure may be used to eliminate this problem byremoving this context switching between drawing/layout adjustment modesand zoom adjustment modes. This can be done in a number of ways.

In one exemplary implementation, the additional interactively controlparameters provided by one user interface sensor provided for by thedisclosure may be used to control the screen zoom of the displayeddiagram, image, etc., allowing the remaining user interface sensor to beused for drawing/layout adjustment.

In another implementation, the application's displayed diagram, image,etc. may be simultaneously displayed in two scales, i.e., macroscopicand microscopic views. In this implementation, one user interface sensormay be used to adjust aspects of the drawing in the macroscopic viewwith an adjustment resolution appropriate for that scale of rendering,while another user interface sensor may be used to adjust aspects of thedrawing in the microscopic view with an adjustment resolutionappropriate for that scale of rendering. Changes made at one scale arereflected in both the macroscopic and microscopic views. This is, in away, similar to the multiple resolution capabilities described earlier,but here each of the user interfaces is uniquely directed towards one orthe other of the macroscopic and microscopic views. The relative sizesof the macroscopic and microscopic views may be fixed or adjusted asrelevant for the application or task as appropriate. It is also notedthat a similar approach could be implemented with a conventional mouseby moving the cursor between the microscopic view window and themacroscopic view window.

3.14 Interactive Document Style Adjustment

In presentations, document layout, publishing layout, website design,and other applications style sheets, master pages, macros, etc. are usedto set a uniform format and appearance of various aspects such asdefault font sizes, font styles, page margins, backgrounds, borders,indentations, list formats, paragraph formats, line widths, FIG.captions, figure borders, multi-column formats, etc. In many situations,details of these style sheets, master pages, macros, etc. may be in needof interactive adjustment as the project evolves. The additionalinteractively control parameters provided by one user interface sensorprovided for by the disclosure may be used to control assignable detailsof these style sheets, master pages, macros, etc. for interactiveadjustment leaving the other user interface free for traditionalediting. In this way rapid interactive adjustment of both editingdetails and details of style sheets, master pages, macros, etc. can befreely interactively commingled without extensive context switching.

4. Concluding Remarks

The present disclosure at its core provides for a wide range of systemsand methods for realizing and applying user interfaces providing, inmany cases, at least four widely-variable simultaneously interactivelyadjustable parameters. In so doing, the disclosure more broadlyencompasses novel user interface structures, metaphors, and applicationsreadily suggested and enabled by the core of the disclosure but whichmay be indeed realized in ways not involving the core of the disclosure.

While the disclosed subject matter has been described in detail withreference to disclosed embodiments, various modifications within thescope of the disclosure will be apparent to those of ordinary skill inthis technological field. It is to be appreciated that featuresdescribed with respect to one embodiment typically may be applied toother embodiments. Therefore, the invention properly is to be construedwith reference to the claims.

The embodiments of the invention in which an exclusive property orprivilege is claimed are defined as follows:
 1. A method for controllingapplications running on a computing device, the method comprising:detecting, by a touch sensor of a user interface device, at least twopoints of contact of at least two contacting fingers on a surface of theuser interface device; providing, to the computing device, firstmultiple-finger touch signals responsive to the at least two contactingfingers on the surface of the user interface device, wherein the firstmultiple-finger touch signals include a parameter representing a spreadbetween the points of contact of the at least two contacting fingers andare used to provide functionality for window selection amongapplications running on the computing device; detecting, by a touchsensor of the user interface device, a gesture on the surface of theuser interface device; providing, to the computing device, gesturesignals responsive to the gesture on the surface of the user interfacedevice, wherein the gesture signals provide functionality forinteracting with at least one of the applications running on thecomputing device; and providing, to the computing device, secondmultiple-finger touch signals responsive to the at least two contactingfingers on the surface of the user interface device, wherein the secondmultiple-finger touch signals include a parameter representing a spreadbetween the points of contact of the at least two contacting fingers andare used to control zoom functionality of at least one of theapplications running on the computing device.
 2. The method of claim 1,wherein the user interface device comprises a touchscreen, and whereinthe surface of the user interface device comprises a surface of thetouchscreen.
 3. The method of claim 1, wherein the secondmultiple-finger touch signals include a varying width representing avarying spread between the points of contact of the at least twocontacting fingers.
 4. The method of claim 1, wherein the gesturesignals provide functionality for horizontal scrolling or verticalscrolling.
 5. The method of claim 1, wherein the gesture signals providefunctionality for help-window navigation.
 6. The method of claim 1,wherein the computing device provides power to the user interface devicevia an electrical connection between the computing device and the userinterface device.
 7. The method of claim 1, wherein the computing deviceis configured to recharge a rechargeable battery of the user interfacedevice via an electrical connection between the computing device and theuser interface device.
 8. A computer system programmed to perform stepscomprising: detecting, by a touch sensor of a user interface device, atleast two points of contact of at least two contacting fingers on asurface of the user interface device; providing, to a computing device,first multiple-finger touch signals responsive to the at least twocontacting fingers on the surface of the user interface device, whereinthe first multiple-finger touch signals include a parameter representinga spread between the points of contact of the at least two contactingfingers and are used to provide functionality for window selection amongapplications running on the computing device; detecting, by a touchsensor of the user interface device, a gesture on the surface of theuser interface device; providing, to the computing device, gesturesignals responsive to the gesture on the surface of the user interfacedevice, wherein the gesture signals provide functionality forinteracting with at least one of the applications running on thecomputing device; and providing, to the computing device, secondmultiple-finger touch signals responsive to the at least two contactingfingers on the surface of the user interface device, wherein the secondmultiple-finger touch signals include a parameter representing a spreadbetween the points of contact of the at least two contacting fingers andare used to control zoom functionality of at least one of theapplications running on the computing device.
 9. The computer system ofclaim 8, wherein the user interface device comprises a touchscreen, andwherein the surface of the user interface device comprises a surface ofthe touchscreen.
 10. The computer system of claim 8, wherein the secondmultiple-finger touch signals include a varying width representing avarying spread between the points of contact of the at least twocontacting fingers.
 11. The computer system of claim 8, wherein thegesture signals provide functionality for horizontal scrolling orvertical scrolling.
 12. The computer system of claim 8, wherein thegesture signals provide functionality for help-window navigation. 13.The computer system of claim 8, wherein the computing device providespower to the user interface device via an electrical connection betweenthe computing device and the user interface device.
 14. The computersystem of claim 8, wherein the computing device is configured torecharge a rechargeable battery of the user interface device via anelectrical connection between the computing device and the userinterface device.
 15. A computer system programmed to perform stepscomprising: detecting, by a touch sensor of a user interface device, atleast two points of contact of at least two contacting fingers on asurface of the user interface device; providing, to a computing device,first multiple-finger touch signals responsive to the at least twocontacting fingers on the surface of the user interface device, whereinthe first multiple-finger touch signals include a parameter representinga spread between the points of contact of the at least two contactingfingers and are used to provide functionality for window selection amongapplications running on the computing device; detecting, by a touchsensor of the user interface device, a gesture on the surface of theuser interface device; and providing, to the computing device, gesturesignals responsive to the gesture on the surface of the user interfacedevice, wherein the gesture signals provide functionality forinteracting with at least one of the applications running on thecomputing device.
 16. The computer system of claim 15, wherein the userinterface device comprises a touchscreen, and wherein the surface of theuser interface device comprises a surface of the touchscreen.
 17. Thecomputer system of claim 15, the steps further comprising providing, tothe computing device, second multiple-finger touch signals, wherein thesecond multiple-finger touch signals include a parameter representing avarying spread between the points of contact of the at least twocontacting fingers on the surface of the user interface device and areused to control zoom functionality of at least one of the applicationsrunning on the computing device.
 18. The computer system of claim 15,wherein the gesture signals provide functionality for horizontalscrolling, vertical scrolling, or help-window navigation.
 19. Thecomputer system of claim 15, wherein the computing device provides powerto the user interface device via an electrical connection between thecomputing device and the user interface device.
 20. The computer systemof claim 15, wherein the computing device is configured to recharge arechargeable battery of the user interface device via an electricalconnection between the computing device and the user interface device.